Trading Accuracy for Size: Online Small SVMs via Linear Independence in the Feature Space

Support Vector Machines (SVMs) are a machine learning method rooted in statistical learning theory. One of their most interesting characteristics is that the solution achieved during training is sparse, meaning that a few samples are usually considered “important” by the algorithm (the so-called support vectors) and give account of most of the complexity of the classification/regression task. It is then crucial to SVMs to keep the number of support vectors as small as possible, since both the training and testing time depend on the size of the training set and the number of support vectors. In fact, in recent literature this has become a key issue in order to speed up SVMs without losing accuracy. In this paper we propose Online Independent Support Vector Machines, an incremental way of reducing the number of support vectors, based upon linear independence in the feature space. Experiments reveal that, at least on the standard tests we have chosen, our machines achieve a dramatic reduction in the number of support vectors without losing accuracy, especially when finite-dimensional kernels are used.